Colorization by Patch-Based Local Low-Rank Matrix Completion

Colorization aims at recovering the original color of a monochrome image from only a few color pixels. A stateof-the-art approach is based on matrix completion, which assumes that the target color image is low-rank. However, this low-rank assumption is often invalid on natural images. In this paper, we propose a patch-based approach that divides the image into patches and then imposes a low-rank structure only on groups of similar patches. Each local matrix completion problem is solved by an accelerated version of alternating direction method of multipliers (ADMM), and each ADMM subproblem is solved efficiently by divide-and-conquer. Experiments on a number of benchmark images demonstrate that the proposed method outperforms existing approaches. Introduction Because of technology limitations, most movies and pictures produced in the last century are monochrome. Colorization is a computer-assisted process that attempts to recover their original colors with some user-provided color pixels (Levin, Lischinski, and Weiss 2004; Luan et al. 2007). However, traditional colorization algorithms are time-consuming, and involve expensive segmentation and tracking of image regions (Markle and Hunt 1987). A seminal work that drastically reduces the amount of manual input is proposed in (Levin, Lischinski, and Weiss 2004). It assumes that neighboring pixels with similar intensities should have similar colors. This is formulated as an optimization problem which minimizes the difference between the color at each pixel and the weighted average of colors from its neighboring pixels. Computationally, it leads to a sparse linear system which can be efficiently solved. However, on complex textures, the underlying local color consistency assumption may fail. In recent years, it is shown that many image objects can be modeled as low-rank matrices. Examples include the dynamic textures (Doretto et al. 2003), and an image set obtained on a convex Lambertian object under different lighting conditions (Basri and Jacobs 2003). Consequently, lowrank modeling has been popularly used in various image analysis tasks, such as face recognition (Candès and Plan 2010; Chen, Wei, and Wang 2012), background removal Copyright c © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. (Candès et al. 2011), video denoising (Ji et al. 2010), and image restoration (Nguyen et al. 2013). Recently, low-rank modeling has also been applied to colorization. Wang and Zhang (2012) used the robust principal component analysis (RPCA) model (Candès et al. 2011), which assumed that the target color image can be decomposed as the sum of a low-rank matrix and a sparse error component. Computationally, this is formulated as a convex optimization problem which can be solved with the alternating direction method of multipliers (ADMM) (Boyd et al. 2011). By combining with the local color consistency approach in (Levin, Lischinski, and Weiss 2004), state-of-theart colorization results are obtained. However, an image matrix is low-rank only if its columns (or rows) are linear combinations of a small subset of columns (resp. rows). This is the case when the image contains many regular patterns, as is commonly found in manmade objects (such as the building in Figure 1(a)). However, it can be a crude approximation on natural images (Figures 1(c) and 1(e)). Often, a significant number of singular values have to be removed before the image is close to lowrank (Liu et al. 2013). On the other hand, a set of similar images are often lowrank (Cai, Candès, and Shen 2010; Candès et al. 2011; Peng et al. 2012). This motivates the usage of a patch-based approach, which has achieved impressive results on various image processing tasks including denoising (Elad and Aharon 2006; Dabov et al. 2007; Mairal et al. 2009a), superresolution (Yang et al. 2010), and inpainting (Mairal et al. 2009b). Specifically, the proposed colorization scheme (i) divides the image into patches; (ii) groups similar patches together; and then (iii) performs colorization on each lowrank patch group. By allowing patches in the same group to borrow strength from each other, better performance can be attained. Besides, even when the individual patches are not low-rank, the whole group is likely to contain shared patterns among patches and thus has a low-rank structure. Computationally, this model leads to a convex optimization problem which can be solved by ADMM. Moreover, by using the `2 loss (instead of the `1 in (Wang and Zhang 2012)) on the reconstruction error, it will be shown that an accelerated version of ADMM (Goldstein, ODonoghue, and Setzer 2014) can be used. We further develop a novel divide-andconquer algorithm so that each ADMM subproblem can be Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence